Projection of synthetic information

ABSTRACT

A real environment during which a feedback between the position of the information and the projection thereof ensues with means for image processing and/or pattern recognition. The feedback ensues in such a manner that the information is positioned at a predetermined position in the real environment.

FIELD OF TECHNOLOGY

The present disclosure relates to a method for projecting syntheticinformation into a real environment, and also to a mobile terminaldevice embodied for this purpose.

BACKGROUND

The projection of synthetic information or, as the case may be,supplementary information leads to an enhancement of the real world.This process is generally referred to as “augmented reality”. By meansof such methods useful information that is necessary for completing aparticular task can be communicated to the user.

Thus, in the course of a maintenance task, for example, the fitter canreceive support by reference to the camera image of a maintenancefacility, with the next instruction step being artificially superimposedat the corresponding position of the equipment in the camera image. Thisis made possible by the projection of the synthetic information into thethree-dimensional model of the maintenance facility based on preciseknowledge of the camera position.

Methods of this type are also employed in the medical domain in supportof the examination of x-ray and ultrasound images, during an operationfor example.

A projected user interface is described in W0-A-0154110. For the purposeof user input a finger on the interface, for example, is detected bymeans of a camera and analyzed in terms of its position relative tointerface surfaces. In order to detect the position the interface hasmarkings for the camera image, as a result of which the interfacesurfaces can be detected in their location in the camera image.

In all known applications relating to the topic of “augmented reality”the resulting image, which is typically a modified camera image, isvisualized on a display. The user himself/herself is thereforeconstantly required to identify the indicated position within themodified image in the real surroundings or environment. If, in a verycomplex installation, an arrow, for example, is drawn in at a particularpoint in the camera image of the installation, it may be very difficultin certain circumstances to locate the relevant corresponding positionin the real installation.

SUMMARY

The present disclosure avoids at least some of those disadvantages andfurther illustrates a method where synthetic information is projecteddirectly into the real environment so that it is visualized at thecorresponding image position.

According to an exemplary embodiment, synthetic information orsupplementary information is projected into the real environment throughthe use of a camera in association with a projection unit. A correctionof the projection position is effected through the implementation of afeedback between the camera image and the projection unit.

Furthermore, a feedback between camera image and projection unit isimplemented in order to compensate for mapping errors.

BRIEF DESCRIPTION OF THE DRAWINGS

The various objects, advantages and novel features of the presentdisclosure will be more readily apprehended from the following DetailedDescription when read in conjunction with the enclosed drawings, inwhich:

FIG. 1 illustrates a mobile telephone MT used to project syntheticinformation into the real environment according to an exemplaryembodiment.

DETAILED DESCRIPTION

Mobile telephone MT shown in FIG. 1 has, on its rear side, a cameramodule K for recording images from the environment, a processor forprocessing the image information (not shown), and a projection unit P,which is arranged for example rotatably on the top of the mobiletelephone. Disposed on the front of the mobile telephone is a displaydevice AE.

The camera image of FIG. 1 is analyzed using digital image processing.Generally this includes pattern recognition methods so that imagedobjects can be recognized and identified, thereby enabling them to beassigned to a model. Thus, for example, information from thethree-dimensional model of an installation is used as a basis forcalculating the position at which, for example, an arrow shown in FIG. 1is projected into the real environment by means of the projection unitP.

The result of the projection is subsequently analyzed in the cameraimage and the position of the arrow corrected if necessary by means of afeedback. As already explained, the image itself can also be corrected,in terms of its size for example.

The method uses a projection unit P and a camera or, as the case may be,a camera module K. Through analysis of the camera image and using a 3Dobject model as a basis, the coordinate at which the syntheticinformation is projected into the real environment by the projector P iscalculated. The user therefore sees the additional synthetic informationdirectly at the object. The information then determines the furtheractivity of the user.

Thus, in the example of the maintenance task an arrow can be projecteddirectly onto the part that is to be replaced, for example. In this casethe user does not need to look at a display or, as the case may be, thedisplay device AE, and also does not have to recognize the designatedpart in the real environment. As a result dealing with augmented realityis made much easier and also more reliable.

In addition, the camera image is constantly used to monitor the resultof the projection using digital image processing methods. The projectionposition and, in a development, also mapping errors due to thethree-dimensional nature of the object can be captured and corrected.Mapping errors result, for example, when an arrow is mapped on aninclined plane. A possible distortion is eliminated by means of thecorrection.

It should be understood that various changes and modifications to thepresently preferred embodiments described herein will be apparent tothose skilled in the art. Such changes and modifications can be madewithout departing from the spirit and scope of the present disclosureand without diminishing its intended advantages. It is thereforeintended that such changes and modifications be covered by the appendedclaims.

1-4. (canceled)
 5. A method for projecting synthetic information into areal environment, comprising: generating a camera image of the readenvironment; assigning the camera image to a model using means for imageprocessing or pattern recognition' calculating a position of theinformation using the model as a basis; projecting the information intothe environment; and establishing a feedback via the camera imagebetween the position of the information in the model and the projectedposition thereof I the camera image such that the information ispositioned in the real environment at a position determined by themodel.
 6. The method as claimed in claim 5, wherein a feedback betweenthe visualization of the information and the projection thereof isestablished using the means for image processing such that mappingerrors in the projection are corrected.
 7. A mobile terminal device,comprising: a camera; a projection unit for projecting syntheticinformation into a real environment; and means for image processing orpattern recognition; wherein a camera image of the real environment isgenerated and assigned to a model using said means; wherein a positionof the information is calculated using the model as a basis and theinformation is projected into the environment, and wherein a feedbackvia the camera image between the position of the information in themodel and the projected position thereof in the camera image isestablished such that the information is positioned in the realenvironment at a position predetermined by the model.
 8. The terminaldevice as claimed in claim 7, wherein a feedback between thevisualization of the information and the projection thereof isestablished using the means for image processing such that mappingerrors in the projection can be corrected.